License

This document is licensed under a Creative Commons Attribution 4.0 International License.

The code is licensed under the MIT License.

The data used is licensed under a Open Data Commons Attribution License.

See the paper’s “Author Contributions” section for details on the contributors of data files.

Metadata

Required libraries and runtime environment description.

This document is versioned in a public git repo: https://github.com/cgranell/apps-treatment-mental-disorders.

This R notebook contains the code to produce the final figures and tables of the paper.

Data

The final number of surveyed papers is 158. For each paper, we have extracted 70 items. Each item is described in the methods section of the paper.

About 71.5% (N=113) of papers reported some kind of assessment, whereas 28.5% (N=45) reported no assessment at all. Looking closer at the type of assessment: 13.9% (N=22) focus on the effect of intervention on clinical symptomology; 32.3% (N=51) report a mix of effect and usability / UX assessment; 25.3% (N=40) focus solely on usability / UX.

Figures and tables

Figure 2: Temporal trend of the number of articles published and assessment type.

It combines two plots. The top one is a stacked bar chart over years, and the value of each group (assessment type) is in absolute counts. The bottom one is a proportional stacked area chart in which the sum of each year is always equal to hundred, and the value of each group (assessment type) is in percentages.

NEW TABLE: Key variables (sample, experimental conditions, resutls, etc) of ‘RCT’ or ‘pilot RCT’ studies only.

NOTE: In response to the comment G.26 ‘Characteristics of included studies’.

NEW TABLE: Evolution of assessment types

NOTE: In response to the comment G.18 “The authors may consider presenting the data in the ‘Evolution of research and assessment’ paragraph using a table instead”. The following table simplifies the last paragragh of the section ‘Evolution of research and assessment’.

Distribution of assessment type. Percentages refer to the total number of studies.
Assessment type N (%) RCT - N (%) PILOT RCT - N (%)
NO ASSESSMENT 45 (28.5%) 0 (0%) 0 (0%)
USABILITY/UX 40 (25.3%) 2 (1.3%) 0 (0%)
EFFECT AND USABILITY/UX 51 (32.3%) 14 (8.9%) 5 (3.2%)
EFFECT 22 (13.9%) 9 (5.7%) 2 (1.3%)

Figure 3: Distribution of articles per mental disorder, categorized according to assessment type. The embedded pie chart shows the proportional distribution of types of assessment over all mental disorders.

The main plot is a stacked barchart colored by assessment type along with a lollipop chart (white circle at the end of each bar) that shows the total percentatge of each mental disorder. Note that we draw two main colors in the legend: purple represents no assessment at all, the range of greens denotes distinct types of assessment. A pie chart is embedded to show the proportional distribution of types of assessment over all mental disorders.

To complement the previous figure, some percentages are discussed in the section ‘Relation between assessment types and mental disorder’ which are included in the following table (that’s not included in the paper).

Mental Disorder Type of assessment Studies (N) Studies (%)
Depressive disorders EFFECT 6 19.4%
EFFECT AND USABILITY/UX 12 38.7%
NO ASSESSMENT 9 29%
USABILITY/UX 4 12.9%
Various disorders EFFECT 2 9.5%
EFFECT AND USABILITY/UX 7 33.3%
NO ASSESSMENT 4 19%
USABILITY/UX 8 38.1%
Anxiety disorders EFFECT 5 27.8%
EFFECT AND USABILITY/UX 4 22.2%
NO ASSESSMENT 8 44.4%
USABILITY/UX 1 5.6%
Substance-related and addictive disorders EFFECT 2 12.5%
EFFECT AND USABILITY/UX 5 31.2%
NO ASSESSMENT 8 50%
USABILITY/UX 1 6.2%
Schizophrenia spectrum and other psychotic disorders EFFECT AND USABILITY/UX 4 26.7%
NO ASSESSMENT 5 33.3%
USABILITY/UX 6 40%
Trauma and stressor-related disorders EFFECT 2 13.3%
EFFECT AND USABILITY/UX 7 46.7%
NO ASSESSMENT 1 6.7%
USABILITY/UX 5 33.3%
Suicidal behavior disorder/nonsuicidal self-injury EFFECT 1 9.1%
EFFECT AND USABILITY/UX 5 45.5%
NO ASSESSMENT 3 27.3%
USABILITY/UX 2 18.2%
Comorbid disorders EFFECT 1 14.3%
NO ASSESSMENT 2 28.6%
USABILITY/UX 4 57.1%
Bipolar and related disorders EFFECT 1 16.7%
EFFECT AND USABILITY/UX 2 33.3%
USABILITY/UX 3 50%
Obsessive-Compulsive and related disorders EFFECT 1 20%
EFFECT AND USABILITY/UX 3 60%
USABILITY/UX 1 20%
Neurodevelopmental disorders EFFECT AND USABILITY/UX 1 25%
NO ASSESSMENT 1 25%
USABILITY/UX 2 50%
Feeding and eating disorders NO ASSESSMENT 1 25%
USABILITY/UX 3 75%
Sleep-wake disorders EFFECT 1 33.3%
NO ASSESSMENT 2 66.7%
Personality disorders EFFECT AND USABILITY/UX 1 100%
Major and Mild Neurocognitive Disorders NO ASSESSMENT 1 100%

NEW TABLE. Breakdown of treatment modalities

NOTE: In respond to the comment F.2 ‘Breakdown of treatment modalities’ to show psychological framework per study (ALL)

NEW DATA: Apps in multiple papers/studies

NOTE: In respond to the comment A.14 ‘Percentage of apps having been studied in multiple articles’

Out of N=158, there are 138 studies with app names. However, there are 108 unique apps. The unique apps follow a long tail distribution, since 89 appear only once, while 19 apps appear in two or more studies, being two studies the most frequent case (14) and apps in 3 or more studies occurs in only 5 cases. In summary, the percentage of apps having been studied in multiple articles is 108 / 19 (13.8%).

Note also that 3 apps are classified in two categories of mental disorders.

Figure 4: Distribution of articles published for the top 6 mental disorders over time.

A line chart to show the number of papers per mental disorder and year.

Table 1: Mental disorders and the studies targeting them: NA = app name not available/not mentioned.

A tabular, compact distribution of apps per mental disorder, grouping the references tha point the same app together. The number(s) in brackets next to the app name is the reference(s) in which the app is mentioned. In the table below, these references are internal identifiers instead.

Compact distribution of papers (apps) per mental disorder ‘NA’ is app not specified/available
Mental Disorder References by app
Major and Mild Neurocognitive Disorders Rico (452)
Personality disorders EMOTEO (106)
Sleep-wake disorders Sleepcare (181), SleepIO (094;103)
Feeding and eating disorders Jorvie (265), Student Bodies–Eating Disorders (SB-ED) (007), Recovery Record (092;460)
Neurodevelopmental disorders iCanLearn (125), LifePal (142), My MFG (304), TimeOut (454)
Obsessive-Compulsive and related disorders Geo-Feedback App (262), GGOC (473), Live OCD Free (044), Mayo Clinic Anxiety Coach (056), RAW HAND (448)
Bipolar and related disorders MyT (042), PRISM (048), SIMPLe (258;226;138;459)
Comorbid disorders CASA-CHESS (297), Enuresis Trainer (290), Learn To Quit (300), SMI-CM (232), Stay Quit Coach (463), Stop-Cannabis (097), NA (086)
Suicidal behavior disorder/nonsuicidal self-injury BackUp; mEMA (453), BeyondNow (409), BRITE (470), iBobbly (124), Virtual Hope Box (VHB) (026;443), NA (275;012), Blue Ice (245;022;030)
Trauma and stressor-related disorders RELAX (301), NA (302), Life Armor; PE Coach; Positive Activity Jackpot; Eventful; Tactical Breather; Virtual Hope Box; Daily Yoga; Simply Yoga (121;037), PE Coach (062;061;069;049), PTSD Coach (160;118;306;015;088;450;478)
Schizophrenia spectrum and other psychotic disorders iCOPE (090), MindFrame (471), movisenseXS (420), RealLife Exp (419), SlowMo (260), TechCare (273), Temstem (418), Actissist (305;440), FOCUS (089;120), Heal Your Mind (087;406), PRIME (117;408)
Substance-related and addictive disorders Drink Less (468), Fit&Sobber (411), Mind the Moment (171), S-Health (437), SEVA (127), SmartQuit (217), Smoke Mind (065), Social-Local-Mobile (430), A-CHESS (013;307), CET Application (014;021), Kick.it (402;413), Smart-T (038;039)
Anxiety disorders Agoraphobia Free; Stress Free (291), Ångesthjälpen (122), AnxietyCoach (192), CBT Assistant (057), Challenger (464), Lantern (025), PsychAssist (286), Public Speech Trainer (PST) (050), SmartCAT (425), GET.ON PAPP (091;105), NA (123;308;027;285;163;457;472)
Various disorders ACT Daily APP (110), FOCUS (414), Headspace (177), iBobbly (029), iCare-stress (433), IntelliCare Suite (131), MoodMission (427), MoodTrainer (185), myCompass (126), PeerTECH App (119), Pocket Skills (436), Sinasprite (431), SmartCAT (282), SPIRIT (140), The Moment (281), TODAY! (294), Wellframe (034), WellWave (477), NA (475), WorkingWell (220;434)
Depressive disorders 7Cups (082), Be Good to Yourself (466), BlueWatch (401), Dcombat (235), Get Happy Program (058), HeadGear (438), iCare Prevent (455), MedLink (152), Mobile Sensing and Support (183), MoodHacker (187), Moodivate (188), MyGamePlan (203), PRIME-D (279), Push-D (407), SocioEmpathy (421), SPSRS (115), SuperBetter (231), The Sound Advice (428), Thought Challenger (114), TODAC TODAC (020), Kokoro-App (035;263;063), NA (051;023;266;071;303;077;432;435)

NEW TABLE: Alternative to TABLE 1.

NOTE: In response to the comment G.26 ‘Characteristics of included studies’. It adds psychological framework data per study to the big table. In production (manual formating), the table can get more compact, all “NO” can be deleted (empty cell instead), and “YES” can be substituted by coloring the cell depending on the psychological approach, as in the example I provided.

Compact distribution of papers (apps) per mental disorder ‘NA’ is app not specified/available
References by app
Psychological framework
Mental Disorder App name Ref CBT WAV TRA POS
Major and Mild Neurocognitive Disorders Rico 452 NO NO NO NO
Personality disorders EMOTEO 106 NO YES NO NO
Sleep-wake disorders Sleepcare 181 YES NO NO NO
SleepIO 094 YES YES NO YES
103 YES NO NO NO
Feeding and eating disorders Jorvie 265 NO YES NO NO
Recovery Record 092 YES YES NO NO
460 YES NO NO NO
Student Bodies–Eating Disorders (SB-ED) 007 YES NO NO NO
Neurodevelopmental disorders iCanLearn 125 YES NO NO NO
LifePal 142 NO NO NO NO
My MFG 304 NO NO NO NO
TimeOut 454 YES NO NO NO
Obsessive-Compulsive and related disorders Geo-Feedback App 262 YES NO NO NO
GGOC 473 YES NO NO NO
Live OCD Free 044 YES NO NO NO
Mayo Clinic Anxiety Coach 056 YES YES NO NO
RAW HAND 448 YES NO NO NO
Bipolar and related disorders MyT 042 YES NO NO NO
PRISM 048 YES NO NO NO
SIMPLe 138 NO NO NO NO
226 NO NO NO NO
258 NO NO NO NO
459 NO NO NO NO
Comorbid disorders CASA-CHESS 297 YES NO NO NO
Enuresis Trainer 290 YES NO NO NO
Learn To Quit 300 YES YES NO NO
SMI-CM 232 NO YES NO NO
Stay Quit Coach 463 YES NO NO NO
Stop-Cannabis 097 YES NO NO NO
NA 086 NO YES NO NO
Suicidal behavior disorder/nonsuicidal self-injury BackUp; mEMA 453 YES NO NO NO
BeyondNow 409 NO NO NO NO
Blue Ice 022 YES YES NO NO
030 YES YES NO NO
245 YES YES NO NO
BRITE 470 YES YES NO YES
iBobbly 124 NO YES NO YES
Virtual Hope Box (VHB) 026 YES NO NO YES
443 YES NO NO NO
NA 012 YES NO NO YES
NA 275 NO YES NO NO
Trauma and stressor-related disorders Life Armor; PE Coach; Positive Activity Jackpot; Eventful; Tactical Breather; Virtual Hope Box; Daily Yoga; Simply Yoga 037 YES NO NO NO
121 YES NO NO NO
PE Coach 049 YES NO NO NO
061 YES YES NO NO
062 YES YES NO NO
069 YES NO NO NO
PTSD Coach 015 YES NO NO NO
088 YES NO NO NO
118 YES NO NO NO
160 YES NO NO NO
306 YES NO NO NO
450 YES NO NO NO
478 YES NO NO NO
RELAX 301 YES NO NO NO
NA 302 YES NO NO NO
Schizophrenia spectrum and other psychotic disorders Actissist 305 YES NO NO NO
440 YES YES NO NO
FOCUS 089 YES NO NO NO
120 YES NO NO NO
Heal Your Mind 087 YES NO NO NO
406 YES NO NO NO
iCOPE 090 YES NO NO NO
MindFrame 471 NO NO NO NO
movisenseXS 420 YES NO NO NO
PRIME 117 YES YES NO NO
408 NO NO NO NO
RealLife Exp 419 YES NO NO NO
SlowMo 260 NO NO NO NO
TechCare 273 YES NO NO NO
Temstem 418 NO NO NO NO
Substance-related and addictive disorders A-CHESS 013 YES NO NO NO
307 NO NO NO NO
CET Application 014 YES NO NO NO
021 YES NO NO NO
Drink Less 468 YES NO NO NO
Fit&Sobber 411 NO NO NO NO
Kick.it 402 YES NO NO NO
413 YES NO NO NO
Mind the Moment 171 YES NO NO NO
S-Health 437 YES NO NO NO
SEVA 127 YES NO NO NO
Smart-T 038 YES NO NO NO
039 YES NO NO NO
SmartQuit 217 NO YES NO NO
Smoke Mind 065 YES NO NO NO
Social-Local-Mobile 430 YES NO NO NO
Anxiety disorders Agoraphobia Free; Stress Free 291 YES NO YES NO
Ångesthjälpen 122 NO YES NO NO
AnxietyCoach 192 YES NO NO NO
CBT Assistant 057 YES YES NO NO
Challenger 464 YES NO NO NO
GET.ON PAPP 091 YES NO NO NO
105 YES NO NO NO
Lantern 025 YES NO NO NO
PsychAssist 286 NO NO YES NO
Public Speech Trainer (PST) 050 NO NO NO NO
SmartCAT 425 YES NO NO NO
NA 027 YES NO NO YES
NA 123 YES YES YES NO
NA 163 YES NO NO NO
NA 285 YES NO NO NO
NA 308 YES NO NO NO
NA 457 YES NO NO NO
NA 472 YES NO NO NO
Various disorders ACT Daily APP 110 NO YES NO NO
FOCUS 414 YES NO NO NO
Headspace 177 NO YES NO NO
iBobbly 029 YES YES NO NO
iCare-stress 433 YES NO NO NO
IntelliCare Suite 131 YES NO NO YES
MoodMission 427 YES YES NO NO
MoodTrainer 185 YES NO NO NO
myCompass 126 YES YES NO YES
PeerTECH App 119 NO NO NO NO
Pocket Skills 436 NO YES NO NO
Sinasprite 431 YES YES NO NO
SmartCAT 282 YES NO NO NO
SPIRIT 140 YES NO NO NO
The Moment 281 YES YES NO YES
TODAY! 294 YES NO YES NO
Wellframe 034 YES NO NO NO
WellWave 477 NO NO NO NO
WorkingWell 220 NO NO NO NO
434 YES NO NO NO
NA 475 YES NO NO NO
Depressive disorders 7Cups 082 NO YES NO NO
Be Good to Yourself 466 YES YES NO NO
BlueWatch 401 YES NO NO NO
Dcombat 235 YES NO NO NO
Get Happy Program 058 YES YES NO NO
HeadGear 438 YES YES NO NO
iCare Prevent 455 YES NO NO NO
Kokoro-App 035 YES NO NO NO
063 YES NO NO NO
263 YES NO NO NO
MedLink 152 YES NO NO YES
Mobile Sensing and Support 183 YES NO NO NO
MoodHacker 187 YES YES NO YES
Moodivate 188 NO YES NO NO
MyGamePlan 203 NO NO NO NO
PRIME-D 279 YES NO NO NO
Push-D 407 YES NO NO YES
SocioEmpathy 421 NO NO NO NO
SPSRS 115 YES NO NO NO
SuperBetter 231 YES YES NO YES
The Sound Advice 428 YES NO NO NO
Thought Challenger 114 YES NO NO NO
TODAC TODAC 020 YES NO NO NO
NA 023 YES YES NO NO
NA 051 YES YES NO NO
NA 071 YES NO NO NO
NA 077 YES NO NO NO
NA 266 YES YES NO NO
NA 303 YES NO NO NO
NA 432 YES NO NO NO
NA 435 YES NO NO NO

Figure 5: Bubble plot representing technology-related dimensions (software features - orange; built-in sensors - green; analytics - blue) versus mental disorders. Bubble size corresponds with amount of articles.

It is a bubble plot that shows the distribution of papers per mental disorder (bubble size) and technology-related characteristics grouped by dimensions (bubble color): software features, built-in sensors, and analytics. Technology-related characteristics are ranked in each dimension.